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ABSTRACT : PROBLEM TO BE SOLVED: To obtain the terminal device In which the Information based 
on the position of the terminal Is superimposed and displayed on the present picture being 
displayed on a screen. 

SOLUTION: A video inputting TV camera 2, a global positioning system antenna for 
position data determination 4 and a transmitter-receiver 3 are provided to a portable 
personal computer 1 . Moreover, a database is provided in the hard disk of the terminal- 
v The present position information of The terminal, which Is obtained through the antenna 4 

) by a control means, is made as a retrieving key and related information is obtained from 

the database and the information is superimposed on the picture and displayed. 
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(54) [Title of the Invention] Terminal device 



(57) [Abstract] 

[ Problem] To produce a terminal 
device in which information 
based on position is 

super impose -displayed on a 
current image displayed on a 
screen - 

[Solving Means] A terminal 
device characterized by the 
ancillary-attachment of an image 
inputting Televieion camera 2, a 
global positioning system 
antenna for position data 
acquisition 4 and a 

communication transmitter- 
receiver device 3 to a portable 
personal computer 1 and, 
furthermore, by a configuration 
' in which a database is held in 
the hard disk of the terminal 
and in which, using a control 
means, current position 

information of the terminal 
device accfuired by the 
employment of the abovementioned 
antenna is used as a search key 
to acquire related information 
from the database which is 
superimposed and displayed on 
the picture image. 




1 portable peroral conv«t«r 

2 Xoao« inputting Television 



> Cnounicaclan tr«o^nifcCCT- 
I 0^3 antenna Cox- position data 
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(Scope of the pat ant claims] 

[Claim 1] Terminal device, characterized by the ancillary- 
attachment of an image inputting Television camera, a 
global positioning system antenna for position data 
acquisition and a communication transmitter -receiver device 
to a portable personal computer and, furthermore, by the 
provision of a database held in the hard disk of the 
terminal device and a control means which displays the 
picture image input from the abovementioned image inputting 
Television camera on the terminal device image of a display 
part of the abovementioned personal computer and, using the 
current position information of the terminal device 
acquired by the employment of the abovementioned global 
positioning system antenna as a search fcey, acquires 
related information from the abovementioned database and 
superimposes and displays this on the abovementioned 
picture image. 

[Claim 2] Terminal device according to Claim 1, 
characterized in that the control means selects and 
displays one or two or more sets of information from among 
near-range information that denotes current position, mid- 
range information that denotes slightly more distant 
information, and far- range information that denotes remote 
information. 

[Detailed Description of the Invention] 

" [00013 

[Field of Industrial Utilization] The present invention 
constitutes a technique that relates to a terminal device 
that is portable and can be moved, that is to say, to a 
mobile terminal device and, furthermore, to a technique 
that relates to a method of display on this mobile terminal 
device . 
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[0002] Moreover, the invention relates not to personal 
computers used in fixed positions such as in offices but 
ra ther to personal comput er s that are u Bed at a pa int of 
destination or while moving. 
(0003} 

[Prior Art] Although portable terminal devices are 

available in the prior art, because position and time and 
so on cannot be automatically acquired using these terminal 
devices when they are to be recorded, this information must 
be input by the individual. 

[0004] In addition, although devices are available that 
connect communication devices to Television cameras or 
still cameras and, although these devices are capable of 
sending picture images acquired at the transmission aide to 
the other aide by way of a network circuit, ancillary 
information such as the position at the time the images 
were taken must be sent separately. This information, oven 
at the transmission side, cannot be recorded and held. 
[00051 Furthermore, although television conference systems 
for which desktop computers axe employed are also 
available, because these systems are premised on the 
employment of a location of use thereof that is fixed, 
similarly to the above -no ted Television camera transmission 
devices, ancillary information such as position is not held 
by the transmission side and, without the necessity for 

said, c anno t b e tr ansmitted either * „ 

[00061 

[Means to Solve the Problems] As the terminal technologies 
of the prior art described above do not comprise a means 
for the acquisition of current position data, information, 
based on position data cannot be displayed on a free 
! terminal device. In addition, because position data cannot 



PAGE 23/53 1 RCVD AT 1 1/18/2004 1:12:50 PM [Eastern Standard Time] k SVR:USPTO-EFXRF-1/0 * DNIS:8729306 1 CSID: * DURATION (mm-ss):14-32 



11/18/2004 THU 13:08 FAX 



©024/053 



5 

be sent to the point of communication, display information 
based on position data cannot be displayed at the reception 
aide* 

[00071 As display based on position of a terminal device 
cannot be per £ orated , there is no method by which, even for 
superimposed -display on a screen, information relating to 
current images input from a Television camera can be 
displayed. 

[0008] The objective of the present invention/ which 
resolves the problem of communication of the geographical 
position of the free terminal device to the transmission- 
side terminal device, ie to provide a terminal device able 
to superimpose-dieplay information baaed on position on to 
a current image displayed on a screen. 

[O00$] 

[Means to Solve the Problems] The principal characterizing 
feature of the present invention is that a Television 
camera and position data acquisition device (global 
positioning system) axe connected to a portable mobile 
terminal device* and that information related thereto is 
superimposed on an image input from the Television camera 
and displayed on the screen. 

[0010] This differs from the technique of the prior art in 
that the image and position data are simultaneously 
acquired, and in that information based on the acquired 
image and p osition d ata _are si mul t a neousl y_ dis played on the 
terminal device. 
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(Embodiment of the Invention] 

[First Embcpdiinont] A detailed description of a first 
embodiment < corresponds to Claim 1) is given below with 
reference to the diagrams . 

[0012] Figure 1 is a schematic diagram of a terminal device 
pertaining to the present invention. The symbol 1 in the 
diagram denotes a portable personal computer and the symbol 
1A denotes a display part thereof. The symbol 2 denotes an 
image inputting Television camera and the symbol 3 denotes 
a communication transmitter -receiver. The symbol 4 denotes 
a position data acquisition global positioning system 
(hereafter referred to as GPS) antenna. It should be noted 
that, as the function of a GPS antenna itself is well 
known, the description thereof has been omitted. 
C0Q13] Figure 2 shows the process flow that follows data 
input. A description will be given of the terminal device 
of Figure 1 with reference to this process flow. By means 
of an image display function of the terminal device, an 
unaltered display 22 of image data 14 input from the image 
inputting Television camera (hereafter referred to simply 
as the Television camera) 2 is performed on the display 
part 1A. in addition, image processing/recognition 
processing IS is performed to extract identifying elements 
from the image and to acquire recognition data 16 . An 
example of recognition data 1 6 is a subject name. A 
database (DB) search processing 17 is implemented using the 
acquired recognition date 16, and a search output 18/ which 
constitutes ancillary information and related information 
of the subject, is acquired, and a display 22 thereof is 
performed. 
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[0014} A DB search processing 20 o£ position data 19 
acquired on the basis of a signal input from the GPS 
antenna for position data acquisition 4 is implemented to 
acquire geographical information, (names of places and so 
on) and related information ("near subject* and so on) 21 
and a display 22 is performed on the screen. The 
geographical information /related information 21 can be used 
as input for the db search processing 17 for specifying the 
recognition data 16 and subject, and it can increase the 
accuracy of the search output 18 acquired thereafter. 
[0015] A DB search - conversion processing 12 is 
implemented on the basis of the contents of reception data 
11 acquired by the communication transmitter-receiver 3 to 
acquire search output - display data 13 - Examples of 
reception data 11 include the transmission- side terminal ID 
number and geographical information, and examples of the 
search output and display data 13 acquired by the 
conversion thereof include the image display adaptor type 
based on the terminal ID and the display position on the 
screen- A display 22 o£ the search output ► display data 13 
is performed on the reception- side terminal device. 
[0016 J The display 22 constitutes a display of the subject, 
and the image data 14, search output 18 and geographical 
information • related information 21 constitute, without 
alteration, the subject for transmission 23. 

[0017 ] in addition , a CPU. a s t he con trol means f or 
implementing the control described above, is ancillary- 
attached to the portable personal computer 1, and the 
display is performed in the display part 1A thereof. 
£0018] [Second Tfrnhodiment ] Next, a description will be 
given of a second eiribodiment of the present invention. 
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f0019] The description provided below pertain to the 
expansion of the processing contents of the display 22 
shown in Figure 2. 

[0020] Although the display 22 is performed on the display 
part 1A of the portable personal computer, this display 
part 1A, as shown in Figure 3, constitutes a configuration 
for display that is divided into lower planes 31 for 
! display of image data 14 and upper planes 32 for display of 

| other data. Although, in Figure 3, the bit planes for image 

display are shown as the lower planes 31 and the bit planes 
for superimpoaed-display of other data are displayed as the 
upper planes 32, this may be reversed. 

[0021] The lower planes 31, which constitute the bit planes 
for image display, are prepared as 1 plane (1 bit, 2 
colours) when the image display is monochrome and, when the 
image display is colour and, depending on the number of 
colours, are prepared as S planes (8 bits, 256 colours) or 
24 planes (24 bits, 16*400,000 colours, full colour). 
[0022] The representation capacity of the- upper planes 32. 
which constitute the bit planes for superimposed-display, 
need only be small in number so they are prepared aa one to 
two planes. For the purpose of simplicity of the following 
description thereof, the number of lower planes 31 shall be 
taken ae 8 and the number of upper planes 32 shall be taXen 
as 1. It should be noted that the planes are theoretical 

_ ^ bit planes and the value of these bit planes is 

superimposed by oar. 

[0023 3 Figure 4 shows an example of a display 22 in which 
the lower planes 31 and upper planes 32 are superimposed. 
[0024] In Figure 4, images from the Television camera 2 are 
input - displayed in the lower planes 31 shown in' Figure 3, 
and information display based on position input from the 
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GPS antenna 4 is input: into the upper planes 32 shown in 
Figure 3 and a display 22 thereof is performed. 
[0025] In Figure 4, the buildings and the road shape in the 
direction of movement of the current position indicate the 
information display based on position input from the GPS 
antenna 4. 

(0026) Figure 5 shows a display 22 different to Figure 4 
(corresponds to Claim 2) . 

[0027} Although, in Figure 5, near-range information that 
denotes current position, mid-range information that 
denotes slightly distant information and far-range 
information that denotes remote information denote the 
information displayed in the upper planes 32, either one or 
two of these three information sets can be selected, it 
should be noted that, even with a normal display part 1A 
that does not comprise upper and lower planes 32, 31, a 
superimposed-display of other data can be performed 
together with the image display. 

[0028] Figure € is a flow diagram of the processing that is 
employed in the embodiments of Figure 4 and Figure 5. It 
should be noted that the symbols CSD to tS8) denote the 
process thereof • 

[0029] In processing (£1) , images are input from the 
Television camera 2 and, in processing <S2>, these images 
are displayed unaltered on the lower planes 31. 

_[00301 Meanwhile, in processing (S3) , data re ception from 

the GPS antenna 4 is simultaneously implemented with the 
input of picture images and, in processing (S4>, a position 
calculation is performed. As . a result, latitude and 
longitude position information can be acquired. In 
processing (S5) , command information such as far-range 
display and near-range display are received and, in 
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processing (S6) , on the basis of this and previous position 
Information and the latest position information, a search 
processing from the database is implemented. In processing 

(S7), the information acquired bb a result of this search 
is used to determine positional arrangement and, in 
processing (SB), is displayed on the upper planes 32. 

[0031] In both processing (52) and processing (S8) the 
images displayed on the upper and lower planes 32, 31 are 
superimposed and can be simultaneously observed by the 
user. 

[0032] Next, a detailed description of the processing 
contents of processing (S6) and processing (£7) of Figure € 
will be given. 

[0033] Figure 7 represents the configuration of the 
database that constitutes the subject for the processing 
that is implemented in the processing (56) of Figure 6 in 
which position data is used as a key to search for related 
information. 

[0034] Registered subjects in the database are registered 
with the assigning of position data 1 (latitude 
information) , position data 2 (longitude information) 
position data 3 (far, mid, near information) , subject 
attributes (road information, building information and so 
on) and display information (text data and diagrammatical 
data and so on) respectively. 

[0 035] A descripti on will be given, with ref ere nce t o 
Figure 8, of the processing implemented in the processing 

(S6> of Figure 6 in which current position data is used as 
a key to search for related information. 

[003S] The following definitions are used in Figure 8. 
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(0037) The syttibol O denotes position based on previously 
acquired position information . In Figure a this is denoted 
as P-l. In addition, the position coordinates are expressed 
as (ac-l. y-lK These may be considered as latitude and 
longitude respectively. 

[Q03B] The synibol @ denotes current position based on the 
latest position information ae<juired- In Figure 8 this is 
denoted as Po. In addition, the position coordinates are 
expressed as (xo, yo) . 

[003 9 J The symbol • denotes position of the registered 
subjects - of which there are a plurality. One thereof, as 
shown in Figure B, is denoted as Pi, and the position 
coordinates thereof are expressed as (xi, yi) - 
[0040] First. a direction of progress vector Vd is 
calculated from the information described above. When Vd = 
(xd, yd) then xd=xo-x-l, yd=yo-y~l. 

[0041] Here, the following processing is ii^plemented on the 
individual registered subjects within the database. In the 
description given below, the processing is described as 
being implemented on an i** number of registered subjects. 
In order to implement the processing on all registered 
subjects the processing should be implement ed increasing i 
from 1 to the final registered subject I * 

[0042] 1) First, a vector Vi between the current position 
** Po and the actual position Pi of the subject is c alculated. 
[0043 T 

[ECfuaxion II 

When Vi- (xdi, ydi) , then 
[0044] 

[Eolation 2 J 

xdi = xi-xo, ydi**yi-yo 
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[0045 ] 2 ) Next r an angle 0 formed between Vd emd Vi is 

calculated. 

[0046} 

[Equation 3] cos 6 = (Vd • Vi)/{|Vd| - |Vi|) 
However , 
[0047] 

(Equation 4] 

(Vd - Vi) = xd*xdi + yd*ydi 

lVd| = Vxd 2 + yd 2 

JVi| a V(xdi*xdi + ydi*ydi) 

[0048> 3) Here, if the field of view angle established in 
advance is taken as a and the vector is larger than the 
value of half this angle, the subject will be deemed not to 
be within the field of view and the following processing 
will not be implemented. 

[0049] That is to say, if 8 > a/2, the following processing 
is skipped- 

[0050] 4) Next, a distance Li, equivalent to the subject 

distance, is obtained. 

[0051] 

[Equation 5] 

Li = |Vi| = ^(xdi*xdi + ydi-ydi) 
5) If Li is larger than the current designated distance 
^ range, for example, than the numeri cal val u e assigned usi ng 
bhe far-range and near-range buttons of Figure 5 , this is 
deemed to be beyond the range of the search subject and the 
following processing will not be implemented. 
[00S2] That is to say, if the numerical value assigned by 
the operation of the user is taken as LI and Li>Ll, thft 
following processing is skipped. 
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[0053] 6) The registered subjects that remain as a result 
of this processing are displayed, and the information in 
the database is transferred to the following display 
processing. 

[0054] A description will- be given, with reference to 
Figure 9, Of a method for determining position when 
information obtained following the search implemented in 
che processing (S7) of Figure 6 i© displayed on the screen. 
C0055] The following definitions are used in Figure 9. 
£0056] The symbol H denotes the length in the vertical 
direction, that is to say, the height, of the screen or the 
window within the screen in which the search results are 
displayed. 

[0057) The symbol L denotes the length in the horizontal 
direction, that is to say, the width, of the screen or the 
window within the screen in which these search results are 
displayed. 

[0058] The symbol l'i constitutes the distance to the right 
side from the centre line of the display screen or the 
width of the window of the display position of the display 
information of the registered subject acquired as a search 
result. 

[0059] The symbol h'i constitutes the distance of this 
display position below the display screen or window. T*he 
display positions l'i, h'i are determined by the following 
proc edure. 

[0060] 1) The horizontal position is established from the 
positional relationship between the current position and 
the search extracted registered subject. tfhat is to say, 
[0061] 

f Equation 6] 

L/2: sin <Jl/2) ^= l'i; sin 8 
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Accordingly, 
[0062] 

[Equation 7] 

l'i = (1/2) sin 9 
2) The vertical position, as one example, is determined by 
the employment of f (x) of the following relationship: 
[0063] 

[Equation 8] 

f(x) = (n/2) arctan(x) 
That is to Bay, 
[0064] 

[Eolation 9] 

h'i = H*f [Li) 

Naturally, it goes without saying that other methods 
besides this method may be employed for determining 
position. 
[0065] 

[Bffect of the invention] As is described above, by 

virtue of the fact that, in the present invention, a 
Television camera and position data acquisition device are 
connected to a mobile terminal device and images input from 
the Television camera are superimposed with information 
related thereto and displayed on a screen and, by virtue of 
the fact chat a search processing is imp lamented on the 
basis of search location information, an information search 
- more — app r o pr i at e — to — the — us«£ — can — be — implemented ■ — This — is_ 
advantageous in that, during searches at this time the user 
need not be concerned about position. 

[0066] In addition, by virtue of the fact that near-range 
information, mid-range information and far-range 
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information can be selected and displayed on the screen, 
the uptake o£ position information is simplified. 
[Brief Description of Diagrams} 

[Figure 1] is a diagram that shows the configuration of a 
first embodiment of a terminal device pertaining to the 
present invention? 

[ Figure 2 ] is a diagram that show3 the pr oces s ing f low 
following input of the range of data of Figure 1; 
[Figure 3] is an explanatory diagram of the configuration 
of the display part of Figure l; 

[Figure 4) is a diagram that shows a display function 
summary of the display part of Figure 1; 

[Figure 53 is a diagram that shows another display function 
summary of the display part of Figure 1/ 

CFigure 63 is a flow diagram for explaining the processing 
flow of the embodiment of Figure A and Figure 5; 
[Figure 7 3 is a diagram for explaining the storage 
configuration of the database in which position data is 
used as a key for searching for related information; 
(Figure 8] is a diagram for explaining the processing in 
which the current position is used as a key for extracting 
registered information from the database; and 
[Figure 9] is a diagram for explaining the display position 
when information acquired following the search processing 
is displayed on the screen. 

[Ejco lanation of Sym bols 1 . 

1 Portable personal computer 
1A Display part 

2 Image inputting Television ca m era 

3 Communication transmitter-receiver 

A GPS antenna for position data acquisition 
11 Reception data 
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12 DB search /conversion processing 

13 Search output/display data 

14 Image data 

15 Image processing /Recognition processing 

16 Recognition data 

17 DB search processing 
IB Search output 

19 Position data 

20 DB search processing 

21 Geographical information ■ Related information 

22 Display 

23 Transmission 

31 Lower planes 

32 Ufc>per planes 
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